Papers with language tasks
Speculative Contrastive Decoding (2024.acl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit exceptional performance in language tasks, yet their auto-regressive inference is limited due to high computational requirements and is sub-optimal due to the exposure bias. |
| Approach: | They propose a decoding approach that leverages predictions from smaller language models to achieve both decoding acceleration and quality improvement. |
| Outcome: | The proposed method achieves both decoding acceleration and quality improvement on four diverse language tasks. |
DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Encoder-decoder transformer models suffer from high inference latency due to auto-regressive decoding . Typically, the decoder takes up most of the latency because of the auto-decoding - a problem that is not solved by the current model. |
| Approach: | They propose an approach to perform Dynamic Early Exit on Decoder to reduce inference latency by 20%-74% by using a multi-exit encoder-decoder transformer model trained with deep supervision. |
| Outcome: | The proposed model reduces inference latency by 20%-74% with comparable or even higher accuracy compared to baseline models. |
Establishing Trustworthiness: Rethinking Tasks and Model Evaluation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language understanding is a multi-faceted cognitive capability, which the Natural Language Processing community has striven to model computationally for decades. |
| Approach: | They propose to rethink what constitutes tasks and model evaluation in NLP and pursue a more holistic view on language, placing trustworthiness at the center. |
| Outcome: | The proposed models are based on generative models and are being deployed in more real-world scenarios, including previously unforeseen zero-shot setups. |
Prompt Engineering a Prompt Engineer (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies indicate that large language models can be meta-prompted to perform automatic prompt engineering, but their potential is limited due to insufficient guidance for complex reasoning in the meta-prompt. |
| Approach: | They propose to infuse three key components into a meta-prompt to guide reasoning . they find prompts that outperform “let’s think step by step” by 6.3% on MultiArith and 3.1% on GSM8K . |
| Outcome: | The proposed method outperforms “let’s think step by step” by 6.3% on MultiArith and 3.1% on GSM8K and outperfies baselines on counterfactual tasks by 6.9%. |
Enhance Robustness of Sequence Labelling with Masked Adversarial Training (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Adversarial training (AT) has shown strong regularization effects on deep learning algorithms by introducing small input perturbations to improve model robustness. |
| Approach: | They propose to use adversarial training to improve robustness from contextual information in sequence labelling tasks by masking or replacing some words in the sentence. |
| Outcome: | The proposed method shows significant improvements on accuracy and robustness of sequence labelling on CoNLL 2000 and 2003 benchmarks. |
KGI: An Integrated Framework for Knowledge Intensive Language Tasks (2022.emnlp-demos)
Copied to clipboard
Md Faisal Mahbub Chowdhury, Michael Glass, Gaetano Rossiello, Alfio Gliozzo, Nandana Mihindukulasooriya
| Challenge: | Existing state-of-the-art retrieval augmented generation models are not available for knowledge-intensive language tasks. |
| Approach: | They propose a retrieval augmented generation system that showcases the latest state-of-the-art retrieval models on knowledge-intensive language tasks. |
| Outcome: | The proposed system is based on the core of the KGI system. |
Neural Machine Translation for Agglutinative Languages via Data Rejuvenation (2025.acl-srw)
Copied to clipboard
| Challenge: | Recent years, advances in Neural Machine Translation (NMT) heavily rely on large-scale parallel corpora. |
| Approach: | They propose to combine fine-grained inactive sample identification with target-side rejuvenation to improve translation quality from agglutinative languages. |
| Outcome: | The proposed framework improves on four low-resource agglutinative language tasks. |
Phylogeny-Inspired Adaptation of Multilingual Models to New Languages (2022.aacl-main)
Copied to clipboard
| Challenge: | Large pretrained multilingual models have delivered promising results due to cross-lingual learning capabilities on a variety of language tasks. |
| Approach: | They propose to use language phylogenetic information to improve cross-lingual transfer by leveraging closely related languages in a structured, linguistically-informed manner. |
| Outcome: | The proposed model significantly improves on the baseline model on languages unseen during training. |
Practical Takes on Federated Learning with Pretrained Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | federated learning with pretrained language models for language tasks entails data privacy constraints when learning from diverse data domains. |
| Approach: | They propose to use pretrained language models to learn from diverse data domains . they elaborate hypotheses over the components in federated NLP architectures based on three tasks . |
| Outcome: | The proposed model can generalize by adapting to the different domains. |
SparkRA: A Retrieval-Augmented Knowledge Service System Based on Spark Large Language Model (2024.emnlp-demo)
Copied to clipboard
Dayong Wu, Jiaqi Li, Baoxin Wang, Honghong Zhao, Siyuan Xue, Yanjie Yang, Zhijun Chang, Rui Zhang, Li Qian, Bo Wang, Shijin Wang, Zhixiong Zhang, Guoping Hu
| Challenge: | Large language models (LLMs) have shown remarkable achievements across various language tasks. |
| Approach: | They propose a scientific literature LLM and a knowledge service system based on it . they collect scientific literature and then pre-train it using autoregressive training . |
| Outcome: | The proposed system provides literature investigation, paper reading, and academic writing functions. |
Unveiling the Generalization Power of Fine-Tuned Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional multitasking abilities, but the comprehensive effects of fine-tuning on the LLMs’ generalization ability are not fully understood. |
| Approach: | They conduct extensive experiments across five distinct language tasks on different datasets to investigate whether fine-tuning affects the generalization ability intrinsic to LLMs. |
| Outcome: | The proposed model can generalize to different domains and tasks by integrating the in-context learning strategy during fine-tuning on generation tasks. |
Mitigating Hallucination by Integrating Knowledge Graphs into LLM Inference – a Systematic Literature Review (2025.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant progress on different language tasks, but they tend to "hallucinate" plausible but factually incorrect answers. |
| Approach: | They propose to integrate knowledge graphs (KGs) into LLM inference to reduce hallucinations by searching online and applying a selection process. |
| Outcome: | The proposed integration improves performance on benchmark datasets and also to mitigate hallucinations. |
LLMs Are Zero-Shot Context-Aware Simultaneous Translators (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing SiMT systems operate on a sentence level, disregarding the context established by previous sentences or the broader context implied by previous words. |
| Approach: | They show that open-source LLMs perform on par with or better than some state-of-the-art baselines in simultaneous machine translation tasks, zero-shot. |
| Outcome: | The proposed models perform on par with or better than state-of-the-art baselines in simultaneous machine translation tasks, zero-shot. |
CLIPText: A New Paradigm for Zero-shot Text Classification (2023.findings-acl)
Copied to clipboard
| Challenge: | Experimental results show that CLIP can be applied to zero-shot text classification tasks. |
| Approach: | They propose a CLIP model for zero-shot text classification that integrates prompt into CLIPText to better derive knowledge from CLIP. |
| Outcome: | The proposed model can be applied to a text-image matching problem and show that it can be used for language tasks. |
ORAL: Prompting Your Large-Scale LoRAs via Conditional Recurrent Diffusion (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to low-rank Adaptation (LoRA) are limited in scalability and controllability. |
| Approach: | They propose a conditional recurrent diffusion framework that generates LoRA parameters directly . they integrate model architecture and textual task specifications to generate task-specific parameters . |
| Outcome: | The proposed framework scales to billions-of-parameter LLMs and maintains controllability. |
Text-to-Code Generation with Modality-relative Pre-training (2024.eacl-long)
Copied to clipboard
| Challenge: | Large pre-trained language models have been applied to programming language tasks with great success, often through further pre-training of a strictly-natural language model. |
| Approach: | They propose to map programming language modalities into the same embedding space by separating embeddable spaces between modality and modality-relative training objectives. |
| Outcome: | The proposed model can be adapted and represented differently depending on which modality they belong to and to the ultimate benefit of the downstream task. |
Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imagination (2022.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale pretrained language models suffer from reporting bias, describing the lack of explicit commonsense knowledge in written text. |
| Approach: | They propose to endow language models with visual imagination capabilities by recalling existing images and synthesizing nonexistent images via text-to-image generation. |
| Outcome: | The proposed model improves the performance of existing language models across a diverse set of language tasks. |
L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Quantization-aware PEFT methods have been developed to reduce memory and computational costs associated with large language models. |
| Approach: | They propose a method that integrates Quantization-Aware Training (QAT) with LoRA to reduce memory overhead and improve model accuracy. |
| Outcome: | The proposed method significantly reduces QAT’s memory overhead while preserving the advantage of QAT in producing fully quantized LLMs with high accuracy. |
Developing and Utilizing a Large-Scale Cantonese Dataset for Multi-Tasking in Large Language Models (2025.findings-emnlp)
Copied to clipboard
Jiyue Jiang, Alfred Kar Yin Truong, Yanyu Chen, Qinghang Bao, Sheng Wang, Pengan Chen, Jiuming Wang, Lingpeng Kong, Yu Li, Chuan Wu
| Challenge: | Cantonese is considered a low-resource language due to the dominance of Mandarin . rich colloquial vocabulary of Cantone, English loanwords, and code-switching characteristics add to the complexity of corpus collection and processing. |
| Approach: | We collect Cantonese texts from open source corpora, Hong Kong-specific forums, Wikipedia . we refine the model through supervised fine-tuning on curated Cantonesian tasks . |
| Outcome: | The model achieves state-of-the-art (SOTA) performance on four Cantonese benchmarks. |
Residual Learning of Neural Text Generation with n-gram Language Model (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that n-gram models can achieve satisfactory performance on a large proportion of testing cases. |
| Approach: | They propose to learn a neural LM that fits the residual between an n-gram LM and the real-data distribution. |
| Outcome: | The proposed model achieves additional performance gains over popular standalone models on three typical language tasks. |
Constructing contrastive samples via summarization for text classification with limited annotations (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Various contrastive learning methods have been developed and lead to state-of-the-art performance in many computer vision tasks. |
| Approach: | They propose a method to construct efficient contrastive samples using text summarization to gain better representations of text classification tasks with limited annotations. |
| Outcome: | The proposed framework gains better representations on text classification tasks with limited annotations and is compared with existing methods on real-world text classification datasets. |
Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Sentence embeddings are useful for language processing tasks, but it is unclear how to produce them from encoder-decoder models. |
| Approach: | They investigate the effects of scaling up sentence encoders to 11B parameters on sentence embeddings from text-to-text transformers (T5) . |
| Outcome: | The proposed models outperform the previous best models on both SentEval and SentGLUE transfer tasks. |
Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work analyzes, quantifies, and mitigates language model biases such as gender, race or religion-related stereotypes in static word embeddings and contextual representations. |
| Approach: | They explain the complexity of gender and language around it and examine how current representations perpetuate harms associated with binary gender. |
| Outcome: | The proposed model and dataset biases perpetuate harms associated with the treatment of gender as binary in English language technologies. |
BBScoreV2: Learning Time-Evolution and Latent Alignment from Stochastic Representation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive generative models are gaining traction in language tasks such as text generation and machine translation. |
| Approach: | They propose a likelihood-based evaluation metric that fits transformer-based model embeddings into a stochastic process and propose it as a probability-based metric. |
| Outcome: | The proposed model embeddings induce a "clustered-to-temporal ordered" mapping of language model representations in high-dimensional space, and this structure enhances performance on tasks such as temporal consistency evaluation and AI-generated content detection. |
Mapping the Course for Prompt-based Structured Prediction (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models have demonstrated strong performance in a wide-range of language tasks without task-specific fine-tuning. |
| Approach: | They combine large language models with combinatorial inference to marry predictive power of LLMs with structural consistency provided by inference methods. |
| Outcome: | The proposed model incorporates symbolic inference to provide consistent and accurate predictions on challenging tasks. |
Automatic Extraction of Language-Specific Biomarkers of Healthy Aging in Icelandic (2024.lrec-main)
Copied to clipboard
| Challenge: | Multiple studies have shown that individuals suffering from AD exhibit difficulties with word retrieval, produce fewer information units and content words, and use more pronouns than healthy age-matched controls. |
| Approach: | They administered three language tasks to participants aged 60–80 to examine the effects of task type and healthy aging on various automatically extracted part-of-speech features in Icelandic. |
| Outcome: | The results show that task type and healthy aging influence language production in Icelandic. |
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2024.acl-long)
Copied to clipboard
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li
| Challenge: | Large language models (LLMs) can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases. |
| Approach: | They propose a bilingual, multi-task benchmark for long context understanding that extends context windows and more sophisticated memory mechanisms to improve models' long context capabilities. |
| Outcome: | The proposed model outperforms open-source models but struggles on longer contexts. |
What do tokens know about their characters and how do they know it? (2022.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models that use subword tokenization schemes can succeed at a variety of language tasks that require character-level information. |
| Approach: | They propose to use word tokenization schemes to probe what word pieces encode . they show that larger models can encode character-level information . |
| Outcome: | The proposed models can encode character-level information and perform better on non-Latin alphabets. |
MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting (2023.eacl-main)
Copied to clipboard
| Challenge: | Large pre-trained models have proved to be remarkable zero- and (prompt-based) few-shot learners in unimodal vision and language tasks. |
| Approach: | They propose to use frozen unimodal models to learn a lightweight mapping between the representation spaces of unimod models using aligned image-text data. |
| Outcome: | The proposed method can generalize to unseen VL tasks from a few in-context examples while training orders of magnitude fewer parameters. |
Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | PIXEL is a vision transformer that has been pre-trained on rendered text . however, it is not able to outperform monolingual subwords like BERT . |
| Approach: | They propose to use PIXEL as a vision transformer to train on rendered text to explore the gap between its visual and linguistic understanding. |
| Outcome: | The proposed model outperforms monolingual subword models in most other contexts, but it lacks the linguistic knowledge to perform in language tasks. |
FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization (2025.findings-emnlp)
Copied to clipboard
Fangxin Liu, Zongwu Wang, Jinhong Xia, Junping Zhao, Shouren Zhao, Jinjin Li, Jian Liu, Li Jiang, Haibing Guan
| Challenge: | Existing methods for quantization of large language models struggle to adapt to dynamic workloads. |
| Approach: | a new framework optimizes the trade-off between inference speed and accuracy . FlexQuant enables fine-grained, layer-wise mixed-precision quantization . |
| Outcome: | a new framework optimizes the trade-off between inference speed and accuracy . it achieves a 1.3 speedup across diverse language tasks with negligible accuracy loss . |
CPL: Counterfactual Prompt Learning for Vision and Language Models (2022.emnlp-main)
Copied to clipboard
Xuehai He, Diji Yang, Weixi Feng, Tsu-Jui Fu, Arjun Akula, Varun Jampani, Pradyumna Narayana, Sugato Basu, William Yang Wang, Xin Wang
| Challenge: | Existing prompt tuning methods tend to learn spurious or entangled representations, leading to poor generalization to unseen concepts. |
| Approach: | They propose a prompt tuning technique that tunes the learnable prompt for pre-trained vision and language models. |
| Outcome: | The proposed method improves few-shot performance on vision and language tasks over existing prompt tuning methods. |
A Closer Look into Mixture-of-Experts in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Mixture-of-experts (MoE) architectures are gaining increasing attention for their unique properties and remarkable performance. |
| Approach: | They propose a mixture-of-experts architecture that allows for model scaling without sacrificing computational efficiency. |
| Outcome: | The proposed model increases model size without sacrificing computational efficiency . the proposed model is modular and can be used by a broad spectrum of practitioners . |
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2022.naacl-main)
Copied to clipboard
| Challenge: | Neural information retrieval (IR) methods encode queries and documents into single vectors, but late interaction models produce multi-vector representations at the granularity of each token. |
| Approach: | They propose a retrieval method that couples an aggressive residual compression mechanism with a denoised supervision strategy to improve the quality and space footprint of late interaction. |
| Outcome: | The proposed retriever improves quality and space footprint of late interaction models while reducing space footprint by 6–10x. |
The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models (2022.emnlp-main)
Copied to clipboard
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, Dan Alistarh
| Challenge: | Pre-trained Transformer models provide robust language representations which can be specialized on various tasks. |
| Approach: | They propose an efficient pruning method based on approximate second-order information that allows pruning weight blocks to be used for pruning. |
| Outcome: | The proposed method is the first to be applied at the BERT scale and significantly pushes the boundaries of the current sparse models with respect to all metrics: model size, inference speed and task accuracy. |
Beyond Scaling: Predicting Patent Approval with Domain-specific Fine-grained Claim Dependency Graph (2024.acl-long)
Copied to clipboard
| Challenge: | Scaling up language models has demonstrated predictable improvement and unprecedented abilities in many language tasks. |
| Approach: | They propose a fine-grained cLAim depeNdency graph that captures the dependencies within the patent data and extends the embedding-based state-of-the-art (SOTA) they then explore prompt-based methods to harness proprietary LLMs' potential, but find the best results close to random guessing, underlining the ineffectiveness of model scaling-up. |
| Outcome: | The proposed graph methods outperform the standard model scaling methods in the patent approval prediction task and show that they are cost-effective. |
HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for detection of hallucinations operate after text generation, making intervention costly and untimely. |
| Approach: | They examine whether hallucination risk can instead be predicted before any token is generated by probing a model's internal representations in a single forward pass. |
| Outcome: | The proposed model can detect hallucinations before token generation, while query-token representations can be more accurate. |
PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers (2024.acl-long)
Copied to clipboard
| Challenge: | Large Multimodal Models excel in natural language and visual understanding but are challenged by challenging tasks such as Knowledge-based Visual Question Answering (KB-VQA). |
| Approach: | They propose a framework for training Large Multimodal Models (LMMs) to perform KB-VQA tasks. |
| Outcome: | The proposed framework is used to train and evaluate multi-modal retrievers. |
Semi-Supervised Lifelong Language Learning (2022.findings-emnlp)
Copied to clipboard
Yingxiu Zhao, Yinhe Zheng, Bowen Yu, Zhiliang Tian, Dongkyu Lee, Jian Sun, Yongbin Li, Nevin L. Zhang
| Challenge: | Existing methods to learn languages only focus on supervised learning, and unlabeled data is underexplored. |
| Approach: | They propose a semi-supervised lifelong language learning setting where a model learns sequentially arriving language tasks with both labeled and unlabeled data. |
| Outcome: | The proposed model outperforms baseline models on various language tasks and is effective and superior to existing models. |
Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural models have established state-of-the-art performance on several NLP benchmarks, but little is understood about the mechanisms by which they operate. |
| Approach: | They examine the probing paradigm through a set of controlled synthetic tasks and show that pretrained word embeddings play a considerable role in encoding these properties rather than the training task itself. |
| Outcome: | The proposed model can encode linguistic properties above chance-level even when distributed in the data as random noise, reversing the interpretation of absolute claims on probing tasks. |
Lexical Generalization Improves with Larger Models and Longer Training (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Excessive utilization of lexical overlap heuristics can lead to failure on challenging inputs. |
| Approach: | They analyze the use of lexical overlap heuristics in natural language inference, paraphrase detection, and reading comprehension using a contrastive dataset. |
| Outcome: | The proposed model can be used to improve performance on a wide range of tasks, but it is often wrong. |
Do LLMs Play Dice? Exploring Probability Distribution Sampling in Large Language Models for Behavioral Simulation (2025.coling-main)
Copied to clipboard
| Challenge: | LLMs are used to emulate sequential decision-making processes of humans . however, their ability to perform probabilistic sampling is limited . |
| Approach: | They propose to use large language models (LLMs) as agents to emulate the sequential decision-making processes of humans represented as Markov decision-makers (MDPs). |
| Outcome: | The proposed models can understand probabilities, but struggle with sampling precision . integrating coding tools can improve sampling precision, but this level of sampling precision still makes it difficult to simulate human behavior as agents. |
Prompting Language Models for Linguistic Structure (2023.acl-long)
Copied to clipboard
| Challenge: | Existing prompting methods can test this hypothesis on autoregressive PLMs. |
| Approach: | They propose a structured prompting approach for linguistic structured prediction tasks that performs zero- and few-shot sequence tagging with autoregressive PLMs. |
| Outcome: | The proposed approach shows that the model can perform few-shot sequence tagging on part-of-speech taging, named entity recognition, and sentence chunking tasks. |
Single Training Dimension Selection for Word Embedding with PCA (D19-1)
Copied to clipboard
| Challenge: | Existing methods for word embeddings are limited by the definition of 'low' dimensionality, which is often used to train word embeds into low dimensional continuous vector space. |
| Approach: | They propose a method to select the number of dimensions for word embeddings using PCA. |
| Outcome: | The proposed method trains one embedding with a generous upper bound (e.g. 1,000) of dimensions and then removes the lesser dimensions one at a time while recording the embeddables’ performance on language tasks. |
Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for language understanding and understanding can be trained to provide contextualized representations of words based on text data. |
| Approach: | They propose a large-scale language VAE model Optimus that is pre-trained on large text corpus and fine-tuned for various language generation and understanding tasks. |
| Outcome: | The proposed model achieves new state-of-the-art on VAE language modeling benchmarks. |
Speech-based Slot Filling using Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have shown an unprecedented ability across various language tasks. |
| Approach: | They propose to use prompts and LoRA fine-tuning to improve slot filling robustness . they propose a linearised knowledge injection scheme to integrate dynamic external knowledge into LLMs. |
| Outcome: | The proposed model improves slot filling with noisy ASR transcriptions with 6.7% and 17.6% absolute SLU-F1 improvements compared to a fully fine-tuned Flan-T5-XL model. |
Self-Adjust Softmax (2025.emnlp-main)
Copied to clipboard
Chuanyang Zheng, Yihang Gao, Guoxuan Chen, Han Shi, Jing Xiong, Xiaozhe Ren, Chao Huang, Zhenguo Li, Yu Li
| Challenge: | Usually, tokens with larger attention scores are important for the final prediction. |
| Approach: | They propose to modify softmax(z) to z softmax and its normalized variant to improve the Transformer attention mechanism by making minor adjustments to the softmax function. |
| Outcome: | The proposed model provides enhanced gradient properties compared to the vanilla softmax function. |
LaMP: When Large Language Models Meet Personalization (2024.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for personalization in large language models are understudied . |
| Approach: | They propose a benchmark for training and evaluating language models for producing personalized outputs using a set of seven personalized tasks . they propose two retrieval augmentation approaches that retrieve personal items from each user profile for personalizing language model outputs. |
| Outcome: | The proposed approach is effective for a set of zero-shot and fine-tuned language models and highlights the impact of personalization in various natural language tasks. |
When the LM misunderstood the human chuckled: Analyzing garden path effects in humans and language models (2025.acl-long)
Copied to clipboard
| Challenge: | Modern Large Language Models (LLMs) have shown human-like abilities in many language tasks, sparking interest in comparing LLMs’ and humans’ language processing. |
| Approach: | They propose to answer two questions: 1. What makes garden-path sentences hard for humans? 2. Do the same reasons make garden- path sentences hard? |
| Outcome: | The proposed models show that humans struggle with specific syntactic complexities, with some models showing high correlation with human comprehension. |
Embedding Hallucination for Few-shot Language Fine-tuning (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained language models can cause severe over-fitting. |
| Approach: | They propose an Embedding Hallucination method which generates auxiliary embedding-label pairs to expand the fine-tuning dataset. |
| Outcome: | The proposed method outperforms current fine-tuning methods in a wide range of language tasks. |
Contrastive Learning for Prompt-based Few-shot Language Learners (2022.naacl-main)
Copied to clipboard
| Challenge: | a recent study has shown that GPT-3 fine-tuning models with limited examples is effective . a contrastive learning framework clusters inputs from the same class under different augmented “views” and repels those from different classes. |
| Approach: | They propose a supervised contrastive framework that clusters inputs from the same class under different augmented "views" they combine a contrastive loss with the standard masked language modeling loss in prompt-based few-shot learners . |
| Outcome: | The proposed framework improves on the state-of-the-art methods in a diverse set of 15 language tasks. |
EIT: Enhanced Interactive Transformer (2024.acl-long)
Copied to clipboard
| Challenge: | Existing multi-view learning models prioritize complementarity while ignoring consensus . EMHA allows for efficient modeling of global dependencies among tokens in parallel . |
| Approach: | They propose an enhanced multi-head self-attention (EMHA) that prioritizes complementarity while ignoring consensus. |
| Outcome: | The proposed method favors consensus among heads by introducing two models . it is superior on a wide range of language tasks with a modest increase in model size . |
Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning (2023.emnlp-main)
Copied to clipboard
Ximing Lu, Faeze Brahman, Peter West, Jaehun Jung, Khyathi Chandu, Abhilasha Ravichander, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, Nouha Dziri, Jillian Fisher, Bill Lin, Skyler Hallinan, Lianhui Qin, Xiang Ren, Sean Welleck, Yejin Choi
| Challenge: | Extreme-scale language models have shown exceptional performance on a variety of language tasks, but the degree of control offered by these models through pure prompting is limited. |
| Approach: | They propose an inference-time policy adapter which tailors a large base model without fine-tuning it. |
| Outcome: | The proposed model outperforms baseline methods on five challenging text generation tasks and even over GPT-4. |
On the Generalization of Training-based ChatGPT Detection Methods (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that training-based methods are ineffective to detect LLM generated texts from unseen tasks or topics which are not collected during training. |
| Approach: | They propose to train classification models to distinguish LLMs from human texts by a distribution shift caused by prompts, text lengths, topics, and language tasks. |
| Outcome: | The proposed methods can detect LLMs from black-box models, but they suffer from distribution shifts due to a wide range of factors, including prompts, text lengths, topics, and language tasks. |
From Heuristic to Analytic: Cognitively Motivated Strategies for Coherent Physical Commonsense Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have shown impressive performance in various language tasks, but are prone to spurious correlations and illusory information. |
| Approach: | They propose to use pre-trained language models to justify decisions with formalized, coherent reasoning chains. |
| Outcome: | The proposed strategies improve coherence of rationalizations yielding state-of-the-art results on Tiered Reasoning for Intuitive Physics (TRIP). |
LLMs + Persona-Plug = Personalized LLMs (2025.acl-long)
Copied to clipboard
Jiongnan Liu, Yutao Zhu, Shuting Wang, Xiaochi Wei, Erxue Min, Yu Lu, Shuaiqiang Wang, Dawei Yin, Zhicheng Dou
| Challenge: | Large language models (LLMs) have demonstrated extraordinary capabilities in natural language understanding, generation, and reasoning. |
| Approach: | They propose a plug-and-play LLM model that embeds a user-specific embedding for each individual by modeling her historical contexts through a lightweight plug-in user embedder module. |
| Outcome: | Experiments on various tasks in the language model personalization (LaMP) benchmark show that the proposed model significantly outperforms existing personalized LLM approaches. |
Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective (2022.emnlp-main)
Copied to clipboard
Ping Yang, Junjie Wang, Ruyi Gan, Xinyu Zhu, Lin Zhang, Ziwei Wu, Xinyu Gao, Jiaxing Zhang, Tetsuya Sakai
| Challenge: | Existing approaches to zero-shot learning are format-agnostic and can address new learning tasks without additional training. |
| Approach: | They propose a new paradigm for zero-shot learning that is format agnostic and compatible with any format and applicable to a list of language tasks. |
| Outcome: | The proposed model shows state-of-the-art performance on several benchmarks and produces satisfactory results on tasks such as text classification and commonsense reasoning. |
Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models (2025.acl-long)
Copied to clipboard
Zhisong Zhang, Yan Wang, Xinting Huang, Tianqing Fang, Hongming Zhang, Chenlong Deng, Shuaiyi Li, Dong Yu
| Challenge: | Large language models have demonstrated remarkable performance across a wide range of language tasks due to their remarkable ability in context modeling. |
| Approach: | They propose to use parallel context encoding to reduce attention entropy by incorporating attention sinks and selective mechanisms to reduce irregular attention . they also propose to incorporate attention sink mechanisms into the parallel encoded context to reduce the irregular attention. |
| Outcome: | The proposed methods lower irregular attention entropy and narrow performance gaps. |
Visual Prompt Tuning for Few-Shot Text Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Existing work on pretraining models for text classification uses image encoders instead of visual prompts. |
| Approach: | They propose a method to deploy large-scale pre-trained models in the prompt-tuning paradigm in few-shot learning. |
| Outcome: | The proposed method outperforms the most recent prompt-tuning methods on five public text classification datasets. |
Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logic (2024.lrec-main)
Copied to clipboard
| Challenge: | Experimental evaluations of large language models demonstrate the efficacy of enhanced reasoning by logic. |
| Approach: | They propose a framework that uses symbolic logic to verify and rectify reasoning steps by steps. |
| Outcome: | The proposed framework improves the zero-shot chain-of-thought reasoning ability of large language models by verifying and rectifying the reasoning steps step by step. |
Uni-Dubbing: Zero-Shot Speech Synthesis from Visual Articulation (2024.acl-long)
Copied to clipboard
Songju Lei, Xize Cheng, Mengjiao Lyu, Jianqiao Hu, Jintao Tan, Runlin Liu, Lingyu Xiong, Tao Jin, Xiandong Li, Zhou Zhao
| Challenge: | Multimodal speech synthesis is a key challenge due to the scarcity of datasets that pair audio with corresponding video. |
| Approach: | They propose a method that incorporates modality alignment during the pre-training phase on multimodal datasets and freezes the video modality extraction component and the encoder module within the pretrained weights. |
| Outcome: | The proposed method achieves a reduced word error rate (WER) of 31.73%, surpassing the previous best of 33.9% with single-modality audio. |
The LLM Language Network: A Neuroscientific Approach for Identifying Causally Task-Relevant Units (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have revealed their potential to perform far more than language processing tasks, showcasing abilities in reasoning and problem-solving. |
| Approach: | They identify language-selective units within 18 popular LLMs using the same localization approach that is used in neuroscience. |
| Outcome: | The proposed method shows that language-selective units are more aligned to brain recordings from the human language system than random units. |
Flexible Generation from Fragmentary Linguistic Input (2022.acl-long)
Copied to clipboard
| Challenge: | dominant paradigm for high-performance models in novel language tasks is direct specialization via training from scratch or fine-tuning large pre-trained models. |
| Approach: | They propose a new model that makes it possible to infer human behavior through basic computational motifs. |
| Outcome: | The proposed model outperforms direct-specialization models in three evaluations and performs comparable to human models. |
EpMAN: Episodic Memory AttentioN for Generalizing to Longer Contexts (2025.acl-long)
Copied to clipboard
Subhajit Chaudhury, Payel Das, Sarathkrishna Swaminathan, Georgios Kollias, Elliot Nelson, Khushbu Pahwa, Tejaswini Pedapati, Igor Melnyk, Matthew Riemer
| Challenge: | Recent advances in Large Language Models (LLMs) have yielded impressive successes on many language tasks, but efficient processing of long contexts remains a significant challenge. |
| Approach: | They propose a method for processing long contexts in an episodic memory module while holistically attending to semantically-relevant context chunks. |
| Outcome: | The proposed method outperforms baseline decoders on multiple long-context recall and question-answering benchmarks on 16k to 256k tokens. |
Text2Chart31: Instruction Tuning for Chart Generation with Automatic Feedback (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets do not cover full range of chart types, such as 3D, volumetric, and gridded charts. |
| Approach: | They propose a hierarchical pipeline and a new dataset for chart generation that leverages the relationships within rich datasets. |
| Outcome: | The proposed method outperforms open-source models and is comparable to state-of-the-art proprietary models in data visualization tasks. |
Are All Spurious Features in Natural Language Alike? An Analysis through a Causal Lens (2022.emnlp-main)
Copied to clipboard
| Challenge: | 'spurious correlations' have been used in NLP to informally denote any undesirable feature-label correlations. |
| Approach: | They formalize this distinction using a causal model and probabilities of necessity and sufficiency, which delineates causal relations between a feature and a label. |
| Outcome: | The proposed model is invariant to the feature, but not sufficient for prediction. |
Large Language Models Can Not Perform Well in Understanding and Manipulating Natural Language at Both Character and Word Levels? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) still exhibit significant deficiencies in basic language understanding and manipulation. |
| Approach: | They propose a bilingual benchmark to assess the performance of Large language models . they use a set of 15 simple text editing tasks to examine their capabilities . |
| Outcome: | The proposed benchmark aims to assess the performance of Large language models in basic language tasks. |
Language acquisition: do children and language models follow similar learning stages? (2023.findings-acl)
Copied to clipboard
| Challenge: | During language acquisition, children follow a typical sequence of learning stages, whereby they first learn to categorize phonemes before they develop their lexicon and eventually master complex syntactic structures. |
| Approach: | They train 48 GPT-2 models from scratch and evaluate their syntactic and semantic abilities at each training step using 96 probes curated from the BLiMP, Zorro and BIG-Bench benchmarks. |
| Outcome: | The proposed model exhibits similar learning trajectories to human children aged between 18 months and 6 years. |
Images in Language Space: Exploring the Suitability of Large Language Models for Vision & Language Tasks (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models have demonstrated robust performance on various language tasks using zero-shot or few-shot learning paradigms. |
| Approach: | They propose to use open-source, open-access language models to make visual input accessible to the model using separate verbalisation models. |
| Outcome: | The proposed model can handle visual input but also require strong reasoning component. |
Domain Adaptation via Prompt Learning for Alzheimer’s Detection (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing attempts to fine-tune pre-trained language models for AD detection are limited by the small and disparate corpora of spoken language. |
| Approach: | They propose to use domain-adaptive prompt fine-tuning to optimize for AD detection by using AD classification loss as the training objective and spoken language corpora from a variety of language tasks. |
| Outcome: | The proposed approach achieves the highest detection rate with the highest-performing prompting paradigm and the domain-adaptive prompt fine-tuning paradigm. |
Layer Duplication in LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Duplicating multihead self-attention layers once or twice yields significant performance boosts. |
| Approach: | They investigate the effect of duplicating multihead self-attention layers in large language models . |
| Outcome: | The proposed method improves performance across a range of language tasks with or without fine-tuning. |
Rubrik’s Cube: Testing a New Rubric for Evaluating Explanations on the CUBE dataset (2025.acl-long)
Copied to clipboard
Diana Galvan-Sosa, Gabrielle Gaudeau, Pride Kavumba, Yunmeng Li, Hongyi Gu, Zheng Yuan, Keisuke Sakaguchi, Paula Buttery
| Challenge: | Large-Language Models (LLMs) are increasingly being used in explanation generation tasks due to their unreliability. |
| Approach: | They propose a rubric and a dataset of 26k explanations written and quality-annotated using the rubric by humans and six open- and closed-source LLMs to test their proposed rubric. |
| Outcome: | The proposed rubric and CUBE dataset focuses on reasoning and language tasks and provides the necessary diversity to test it. |
ApiQ: Finetuning of 2-Bit Quantized Large Language Model (2024.emnlp-main)
Copied to clipboard
| Challenge: | Memory-efficient finetuning of large language models (LLMs) has attracted huge attention with the increasing size of LLMs due to the constraints posed by GPU memory limitations and the effectiveness of these methods compared to full finetune. |
| Approach: | They propose a memory-efficient finetuning framework called ApiQ to restore lost information from quantization by initializing LoRA components and quantizing weights of LLMs. |
| Outcome: | The proposed framework maintains the original LLM’s activation precision while mitigating error propagation from shallower into deeper layers. |
An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Multimodal Models (LMMs) have shown impressive generalization ability on vision and language tasks, but their spatial understanding is under-explored. |
| Approach: | They construct a VQA dataset to analyze LMMs' spatial reasoning capabilities. |
| Outcome: | The proposed model is stronger at basic object detection than complex spatial reasoning. |
FiRST: Finetuning Router-Selective Transformers for Input-Adaptive Latency Reduction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to improve latency via skipping layers have limitations . fiRST is a model-agnostic framework that reduces inference latency while maintaining quality . |
| Approach: | They propose a model-agnostic framework that skips transformer layers during decoding . it is fully compatible with KV caching, enabling faster decoding while maintaining quality . |
| Outcome: | a new framework reduces inference latency by using layer-specific routers to skip transformer layers during decoding. |
Grounding Multilingual Multimodal LLMs With Cultural Knowledge (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new data-centric approach could address cultural gaps in multimodal large language models . despite being trained on billions of image-text pairs, today's models are biased towards English and Western data. |
| Approach: | They propose a data-centric approach that directly grounds MLLMs in cultural knowledge. |
| Outcome: | The proposed approach outperforms open-source models on cultural-focused benchmarks without degrading results on mainstream vision–language tasks. |
To Learn or Not to Learn: Replaced Token Detection for Learning the Meaning of Negation (2024.lrec-main)
Copied to clipboard
| Challenge: | State-of-the-art language models perform well on a variety of language tasks, but struggle with understanding negation cues in tasks like natural language inference (NLI). |
| Approach: | They propose a new learning strategy for negation building on ELECTRA’s replaced token detection objective. |
| Outcome: | The proposed approach leads to substantial gains on a variant of RTE with additional negation. |
Context-Agent: Dynamic Discourse Trees for Non-Linear Dialogue (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to managing non-linear dialogue flow are misaligned with the intrinsically hierarchical and branching structure of natural discourse. |
| Approach: | They propose a framework that models multi-turn dialogue history as a dynamic tree structure. |
| Outcome: | The proposed framework enhances task completion rates and improves token efficiency across various LLMs. |
Entailed Between the Lines: Incorporating Implication into NLI (2025.acl-long)
Copied to clipboard
Shreya Havaldar, Hamidreza Alvari, John Palowitch, Mohammad Javad Hosseini, Senaka Buthpitiya, Alex Fabrikant
| Challenge: | True Emotions, social cues, insults, and a myriad of other messages are conveyed implicitly, often even more so than explicitly. |
| Approach: | They propose a dataset to help LLMs understand implied entailment . |
| Outcome: | The proposed dataset enables LLMs to understand implied entailment and can generalize this understanding across datasets and domains. |
CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large vision-language models have shown impressive ability in various language tasks, especially with their emergent in-context learning capability. |
| Approach: | They propose a causal reasoning benchmark for multi-modal in-context learning from large vision-language models that incorporates visual inputs. |
| Outcome: | The proposed model outperforms existing models on three visual causal reasoning tasks and demonstrates their strengths and weaknesses. |
When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large vision and language models have demonstrated remarkable performance in visual question answering tasks. |
| Approach: | They introduce a framework to optimize L-VLMs by leveraging unlabeled images . they conduct extensive experiments on four diverse VQA benchmarks . |
| Outcome: | The proposed framework improves L-VLMs on four visual question answering benchmarks. |